Transitions between body-locked and world-locked augmented reality

ABSTRACT

Various embodiments relating to controlling a see-through display are disclosed. In one embodiment, virtual objects may be displayed on the see-through display. The virtual objects transition between having a position that is body-locked and a position that is world-locked based on various transition events.

BACKGROUND

Various technologies may allow a user to experience a mix of real andvirtual reality. For example, some display devices, such as various headmounted display (HMD) devices, may comprise a see-through display thatallows superposition of virtual objects over a real-world environment.The virtual objects may appear integrated with the real-worldenvironment when viewed by a wearer of the HMD device through thesee-through display. Such a relationship between the virtual objects andthe real-world environment may be referred to as augmented reality.

SUMMARY

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter. Furthermore,the claimed subject matter is not limited to implementations that solveany or all disadvantages noted in any part of this disclosure.

Various embodiments relating to controlling a see-through display aredisclosed. In one embodiment, virtual objects may be displayed via thesee-through display. The virtual objects transition between having aposition that is world-locked and a position that is body-locked basedon various transition events. A world-locked position appears to befixed relative to real-world objects viewable through the see-throughdisplay and the world-locked position of each virtual object appears tobe moveable relative to a wearer of the see-through display. Abody-locked position appears to be fixed relative to the wearer of thesee-through display and the body-locked position of each virtual objectappears to be moveable relative to the real-world objects.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1 and 2 show a plurality of virtual objects with a world-lockedposition in accordance with an embodiment of the present disclosure.

FIGS. 3 and 4 show a plurality of virtual objects with a body-lockedposition in accordance with an embodiment of the present disclosure.

FIG. 5 shows a plurality of virtual objects as volumetric holograms withworld-locked positions in accordance with an embodiment of the presentdisclosure.

FIG. 6 shows a plurality of virtual objects displayed in a screenviewport with a body-locked position in accordance with an embodiment ofthe present disclosure.

FIG. 7 schematically shows an example tracking system of a head mounteddisplay device in accordance with an embodiment of the presentdisclosure.

FIG. 8 shows an example method for controlling a see-through display inaccordance with an embodiment of the present disclosure.

FIG. 9 shows an example of a head-mounted display device in accordancewith an embodiment of the present disclosure.

FIG. 10 schematically shows an example of a computing system inaccordance with an embodiment of the present disclosure.

DETAILED DESCRIPTION

The present description relates to controlling a see-through display ofa head mounted display (HMD) device to provide an immersive augmentedreality experience that has consistent quality of operation. Moreparticularly, the present description relates to controlling asee-through display device to create a single experience whileseamlessly transitioning between two different tracking systems that areassociated with two different display modes.

In the first display mode, one or more virtual objects may be displayedon the see-through display with world-locked positions. A world-lockedposition of a virtual object appears to be fixed relative to real-worldobjects viewable through the see-through display and the world-lockedposition of each virtual object appears to be moveable relative to awearer of the see-through display. The first display mode may beassociated with a first tracker that estimates a location and anorientation of the HMD device in six degrees of freedom (e.g., x, y, z,pitch, roll, yaw). For example, the estimate in six degrees of freedommay be determined using information from a combination of opticalsensors and other pose sensors that do not rely merely on opticalinformation, such as accelerometers, gyroscopes, magnetometers, or othersensors used in deduced reckoning of a current position/orientation byusing one or more previously determined positions/orientations and knownor estimated changes over designated time periods. The optical sensorsmay provide feedback of optical features of a real-world environment inwhich the HMD device is located. Due to the reliance on light conditionsand optical features of the real-world environment to provide feedback,the optical sensors may be complex and as a result may not always bereliable. However, the augmented reality experience provided by thefirst display mode may be extremely rich and compelling when conditionsare suitable to provide accurate feedback.

If conditions are not suitable to consistently display virtual objectswith world-locked positions in the first display mode, then the HMDdevice may transition to operation in the second display mode. In thesecond display mode, one or more virtual objects may be displayed on thesee-through display with body-locked positions. A body-locked positionof a virtual object appears to be fixed relative to the wearer of thesee-through display and the body-locked position of each virtual objectappears to be moveable relative to the real-world objects. The seconddisplay mode may be associated with a second tracker that operates inparallel with the first tracker. The second tracker estimates anorientation of the HMD device in three degrees of freedom (e.g., pitch,roll, yaw). For example, the estimate in three degrees of freedom may bedetermined using information from pose sensors that do not rely onoptical feedback. Accordingly, the second tracker may operateconsistently during conditions in which the first tracker may provide adegraded augmented reality experience.

By using multiple tracking technologies built into a single HMD device,a high quality augmented reality experience may be consistently providedvia transitions between display modes even as environmental conditionschange. In other approaches, when environmental conditions do not offera single tracker sufficient data to support tracking and mapping, suchdevices simply fail to provide a consistent high quality augmentedreality experience.

FIGS. 1-6 show different scenarios of virtual objects being displayed ona see-through display 100 of a HMD device 102 from a perspective of awearer 104 of the HMD device. More particularly, the different scenariosshows the HMD device operating in different display modes based onconditions of a real-world environment. FIGS. 1 and 2 show an examplescenario where conditions of a real-world environment 106 are suitablefor the HMD device 102 to operate in a first display mode where aplurality of virtual objects (e.g., video slates 1, 2, 3) are displayedon the see-through display 100 with world-locked positions. Inparticular, the real-world environment 106 is a room having variousoptical features, such as a table, doorway, etc. that may providesuitable feedback to provide accurate tracking and mapping of thelocation and the orientation of the HMD device in six degrees offreedom. Such tracking in six degrees of freedom allows the video slatesto have world locked positions in the real-world environment.

In FIG. 1, a first video slate 108 appears to be hanging on a left-sidewall 110 of the room 106 relative to the location and orientation of thewearer 104 of the HMD device 102. A second video slate 112 appears to behanging on a front facing wall 114 of the room 106 relative to thelocation and orientation of the wearer 104 of the HMD device 102. Athird video slate 116 appears to be hanging on a right-side wall 118relative to the location and orientation of the wearer 104 of the HMDdevice 102.

In FIG. 2, the wearer 104 of the HMD device 102 has rotated to the left,so that he/she is facing the wall 110 and the location and theorientation of the HMD device changes. However, since the HMD device isoperating in the first display mode, and the video slates haveworld-locked positions, the first video slate 108 appears to be fixed onthe wall 110 relative to the other real-world objects viewable throughthe see-through display. Likewise, the second video slate 112 appears tobe fixed on the wall 114 relative to the other real-world objectsviewable through the see-through display. Correspondingly, theworld-locked position of each of the first and second video slatesappears to move relative to the wearer of the see-through display.

It will be understood that the HMD device may display any suitablenumber of video slates as virtual objects having world-locked positionson the see-through display. It will be understood that the video slatesmay present any suitable video or other images. For example, the videoslates may present one or more web pages, email applications, weatherreports, television, news, photographs, etc.

Furthermore, it will be understood that the plurality of video slatesmay be located in any suitable world-locked position within the room. Insome embodiments, the plurality of video slates may be displayedaccording to a predefined visual layout. In some embodiments, thepredefined visual layout may be location-specific. For example, opticalfeedback of the tracking system may be used to recognize that thereal-world environment is a particular location, such as the wearer'soffice. The plurality of video slates may be placed in particularlocations within the office according to the location-specific visuallayout. On the other hand, if a real-world environment is unrecognized,the plurality of video slates may be placed according to a defaultvisual layout. In other embodiments, the visual layout may be locationindependent, and the video slates may be displayed in the sameworld-locked positions in any location while operating in the firstdisplay mode.

FIGS. 3 and 4 show an example scenario where conditions of thereal-world environment change causing a transition event in which theHMD device 102 switches from operation in the first display mode tooperation in the second display mode. In this case, the transition eventoccurs when the wearer of the HMD device moves through the doorway ofthe room 106 shown in FIGS. 1 and 2 and into a hallway 300. For example,the transition event may occur based on the estimate of the location andthe orientation of the HMD device in six degrees of freedom not meetinga quality metric. In one example, sensor information used to perform theestimate may not be available. In another example, the estimate may nothave sufficient confidence.

In the second display mode, the plurality of video slates may bedisplayed on the see-through display 100 with body-locked positions. Forexample, the plurality of video slates may appear in a sphericalworkspace layout where the wearer 104 of the HMD device 102 stands inthe center of the sphere and the plurality of video slates surroundhim/her. In some embodiments, the plurality of video slates may bearranged according to a predefined visual layout. In one particularexample, a news application could be displayed in the first video slateon the left-side, an email application could be displayed on in thesecond video slate in the middle, and a stock ticker application couldbe displayed in the third video slate on the right-side.

In some embodiments, the virtual objects may visually transition over aseries of image frames from the world-locked position to the body-lockedposition via an animation. For example, when the wearer leaves the roomand enters the hallway, the video slates could appear to suddenly flyoff the world-locked positions on the walls and transition tobody-locked positions in the spherical work space. The flying animationmay appear to the wearer of the HMD device like a natural action thateases the transition in display modes.

In FIG. 3, the wearer 104 of the HMD device 102 is standing at an end ofthe hallway 300, next to the doorway from the room 106. The plurality ofvideo slates is displayed on the see-through display 100 withbody-locked positions that appear to be fixed relative to the wearer andmoveable relative to real-world objects, such as the doorway.

In FIG. 4, the wearer 104 of the HMD device 102 has walked down to theother end of the hallway 300 and is standing in front of anotherdoorway. Since the plurality of video slates are displayed on thesee-through display 100 with body-locked positions, the plurality ofvideo slates stay fixed relative to the wearer 104 even as he/she walksdown the hallway. In other words, the slates move with the wearer. Theplurality of video slates may be displayed on the see-through display100 with body-locked positions until a transition event occurs thatcauses a switch to the first display mode, at which point the pluralityof video slates may transition to world-locked positions. For example,when tracking in six degrees of freedom becomes available again, theplurality of slates could visually transition from the body-lockedpositions and attach to walls around the wearer of the HMD device in apredefined visual layout. The visual transition may include ananimation, such as the visual slates appearing to fly out from thespherical workspace to the wall and enlarging along the way so thatcontent presented in the video slates can still be seen in theworld-locked position.

FIGS. 5 and 6 show an example scenario where a plurality of virtualobjects are shown in a 3D view in the first display mode and thenvisually transition to a 2D view in the second display mode. In FIG. 5,the HMD device 102 is operating in the first display mode in which aplurality of virtual objects in the form of volumetric holograms (e.g.,3D meshes) 500 may be displayed on the see-through display 100 in a 3Dview. The volumetric holograms may have world-locked positions in thereal-world environment (e.g., a room) 502 and may appear to occupy avolume within the real-world environment.

In FIG. 6, the wearer 104 of the HMD device 102 has walked through thedoorway of the room 502 and into a hallway 504, which causes atransition event to occur in which the HMD device 102 switches from thefirst display mode to the second display mode. In response to thetransition event, the volumetric meshes visually transition from the 3Dview to a 2D view. In particular, the volumetric meshes may collapse toa 2D view of a video slate 506. The video slate may be a screen viewportof a virtual world that the virtual objects inhabit. The video slate 506may have a body-locked position and the virtual objects may move withinthe virtual world depicted in the video slate.

In some embodiments, in response to the transition event, the HMD devicemay capture an image of the real-world environment and display the imagein the video slate. Further, the virtual objects could be shown in theircorresponding real-world positions in the image from the last knownpoint of view before the transition event occurred.

Further, when a later transition event occurs (e.g., when tracking insix degrees of freedom becomes available again), the video slate couldexpand to refill the room (or another current real-world location of theHMD device), and the images of the virtual objects in the video slatecould expand to volumetric holograms in the real-world environment withworld-locked positions.

FIG. 7 schematically shows an example tracking system 700 of a HMDdevice in accordance with an embodiment of the present disclosure. Thetracking system 700 may be configured to select an appropriate displaymode for the HMD device based on conditions of the real-worldenvironment. The tracking system 700 may include a plurality of sensors702 that provide location and orientation information 704 in six degreesof freedom (e.g., x, y, z, pitch, roll, yaw). For example, the pluralityof sensors may include optical sensors and pose sensors. The posesensors may not rely on optical information to determine an orientation.Non-limiting examples of pose sensor include accelerometers, gyroscopes,magnetometers, or other sensors used in deduced reckoning of a currentposition/orientation. The sensor information may be sent to two separatetracking filters that operate in parallel. As discussed in more detailbelow with respect to FIG. 9, the sensors 702 may include any suitablenumber and/or combination of sensors for determining a position,orientation, and/or other movement characteristics of the HMD device inup to six degrees of freedom.

The first tracking filter 706 may be configured to output an estimate710 of a location and an orientation of the HMD device in six degrees offreedom based on the sensor information 704. The estimate 710 of thefirst tracking filter 706 represents highly precise and accuraterotational and translational pose data that typically relies on visiblelight or other optical information. In particular, a translationalpose/location of the HMD device may be estimated relative to otherobjects in a real-world environment based on optical feedback of theenvironment. Further, the real-world environment may be mapped to avirtual model to determine the location of the HMD device relative toother real-world objects. Further, in some embodiments, the opticalinformation may be used in combination with information from posesensors to estimate the rotational pose/orientation of the HMD device.Due to the reliance on visible light, operation of the first trackingfilter may be highly dependent on environmental conditions of thephysical environment in which the HMD device is located.

The second tracking filter 708 may be configured to output an estimate712 of an orientation of the HMD device in three degrees of freedombased on the sensor information 704 of pose sensors. In someembodiments, the estimate of the second tracker may be based sensorinformation from various sensors that do not rely on visible light orother optical information.

The estimate 710 of the location and the orientation of the HMD devicein six degrees of freedom and the estimate 712 of the orientation of theHMD device in three degrees of freedom may be provided to a qualitymonitor 714. The quality monitor 714 may be configured to determinewhether the estimates 710 and 712 meet quality metrics for accuracy andavailability. For example, the quality metric may include comparing theestimates to expected values to determine whether the estimates arewithin a designated accuracy threshold. If the estimates are within theaccuracy threshold, then the estimates may meet the quality metrics.Otherwise, the estimates may not meet the quality metrics. In anotherexample, the quality monitor may determine whether sensor information isreceived from particular sensors used to produce the estimates, and ifthe sensor information is not available than the estimates may not meetthe quality metric.

It will be understood that it may be determined whether the estimatesmeet the quality metrics in any suitable manner without departing fromthe scope of the present disclosure. For example, the quality monitormay be responsible for monitoring tick-by-tick sensor information inputinto the tracking system as well as assessing the estimates over longerperiods to determine a sufficient degradation in quality. If trackingquality degrades either sufficiently or consistently at a duration thatis perceivable by a wearer of the HMD device, the quality monitor canraise a degradation in quality event that causes a transition in displaymodes.

The quality monitor 716 may output tracking quality information to adisplay mode selector 718. For example, the tracking quality informationmay include determinations of whether each of the estimates 710 and 712are accurate or meet a quality metric. The display mode selector 718 mayselect a display mode 720 for operation based on the tracking qualityinformation. In particular, the display mode selector 718 may selectoperation in a world-locked display mode in which virtual objects aredisplayed on the see-through display with a world-locked position if theestimate of the location and orientation of the HMD device in sixdegrees of freedom meets the quality metric. Otherwise, the display modeselector selects operation in a body-locked display mode in whichvirtual objects displayed on the see-through display with a body-lockedposition. In some embodiments, the display mode selector may select adisplay mode based on conditions and/or events other than based on thequality of the estimates. For example, the display mode selector mayswitch display modes responsive to a transition event, such as a wearerinitiated switch in operation.

In some embodiments, the tracking system may be implemented in hardware,such as a processing pipeline including various logical blocks or pipestages. In some embodiments, the tracking system may be implemented assoftware instructions executed by a processor. In some embodiments, thetracking system may be implemented as a combination of hardware andsoftware.

FIG. 8 shows an example method for controlling a see-through display inaccordance with an embodiment of the present disclosure. For example,the method 800 may be performed by the HMD device 900 shown in FIG. 9.In another example, the method 800 may be performed by the computingsystem 1000 shown in FIG. 10.

At 802, the method 800 may include displaying via the see-throughdisplay one or more virtual objects with a world-locked position. Theworld-locked position of each virtual object may appear to be fixedrelative to real-world objects viewable through the see-through displayand the world-locked position of each virtual object may appear to bemoveable relative to a wearer of the see-through display.

At 804, the method 800 may include determining whether a transitionevent occurs. The transition event may include a transition fromtracking a location and an orientation of the see-through display in sixdegrees of freedom to tracking the orientation of the see-throughdisplay in three degrees of freedom. For example, the transition eventmay occur or the transition may be performed based on the location andorientation of the head-mounted display device in six degrees of freedomnot meeting a quality metric, as discussed above. If it is determinedthat there is a transition event, then the method 800 moves to 806.Otherwise, the method 800 returns to 802.

In some embodiments, at 806, the method 800 may include visuallytransitioning the one or more virtual objects over a series of imageframes from the world-locked position to the body-locked position via ananimation.

In one particular example, a plurality of virtual objects in the form ofvideo slates in world-locked positions may appear hanging on differentwalls of a room. The wearer of the see-through display may leave theroom for another location in which tracking in six degrees of freedom isnot available, and thus causes a transition event. Responsive to thetransition event, the plurality of video slates may transition intobody-locked positions appearing as a spherical workspace floating aroundthe wearer of the see-through display. In particular, the plurality ofvideo slates may appear to fly off of the different walls and into apredefined layout in the spherical workspace via an animation. In otherwords, as the wearer leaves the room the slates go with the wearer.

In some embodiments, at 808, the method 800 may include visuallytransitioning the one or more objects from a 3D view of the one or morevirtual objects to a 2D view of the one or more virtual objects.

In one particular example, a plurality of virtual objects in the form ofvolumetric holograms (e.g., 3D meshes) in world-locked positions mayappear located throughout a room. The wearer of the see-through displaymay leave the room for another location in which tracking in six degreesof freedom is not available, and thus causes a transition event.Responsive to the transition event, the plurality of volumetricholograms may collapse to a video slate in a body-locked position via ananimation. The video slate may be a screen viewport (e.g., 2D view) intoa virtual world of the virtual objects. The video slate in thebody-locked position may go with the wearer when the wearer leaves theroom.

At 810, the method 800 may include displaying via the see-throughdisplay the one or more virtual objects with a body-locked position. Thebody-locked position of each virtual object may appear to be fixedrelative to the wearer of the see-through display and the body-lockedposition of each virtual object may appear to be moveable relative tothe real-world objects.

At 812, the method 800 may include determining whether a transitionevent occurs. The transition event may include a transition fromtracking the orientation of the see-through display in three degrees offreedom to tracking the location and the orientation of the see-throughdisplay in six degrees of freedom. For example, the transition event mayoccur or the transition may be performed based on the location andorientation of the head-mounted display device in six degrees of freedommeeting a quality metric, as discussed above. If it is determined thatthere is a transition event, then the method 800 moves to 814.Otherwise, the method 800 returns to 810.

In some embodiments, at 814, the method 800 may include visuallytransitioning the one or more virtual objects over a series of imageframes from the body-locked position to the world-locked position via ananimation.

In one particular example, a plurality of virtual objects in the form ofvideo slates in body-locked positions may appear in a sphericalworkspace around the wearer of the see-through display. The wearer ofthe see-through display may enter a room from another location thatcauses tracking in six degrees of freedom to become available, and thuscauses a transition event. Responsive to the transition event, theplurality of video slates may transition into world-locked positions ondifferent walls of the room. In particular, the plurality of videoslates may appear to fly from the spherical workspace to the differentwalls via an animation.

In some embodiments, at 816, the method 800 may include visuallytransitioning the one or more objects from a 2D view of the one or morevirtual objects to a 3D view of the one or more virtual objects.

In one particular example, a plurality of virtual objects may appear tobe in a video slate that is a body-locked position. The video slate maybe a screen viewport (e.g., 2D view) into the virtual world of thevirtual objects. The video slate in the body-locked position may go withthe wearer wherever the wearer goes. The wearer of the see-throughdisplay may enter a room from another location that causes tracking insix degrees of freedom to become available, and thus causes a transitionevent. Responsive to the transition event, the plurality of virtualobjects in the video slate may expand to volumetric holograms inworld-locked positions via an animation. The volumetric holograms inworld-locked positions may appear located throughout the room.

At 818, the method 800 may include displaying via the see-throughdisplay the one or more virtual objects with a world-locked position.The world-locked position of each virtual object may appear to be fixedrelative to real-world objects viewable through the see-through displayand the world-locked position of each virtual object may appear to bemoveable relative to a wearer of the see-through display.

The above described method may be performed to provide a rich andimmersive augmented reality experience with virtual objects inworld-locked positions via tracking in six degrees of freedom andgracefully transitioning to providing virtual objects in body-lockedpositions via tracking in three degrees of freedom. Such transitions maybe performed in order to mitigate degradations otherwise experiencedwhen tracking in six degrees of freedom does not meet quality metrics orotherwise becomes unavailable.

With reference now to FIG. 9, one example of an HMD device 900 in theform of a pair of wearable glasses with a transparent display 902 isprovided. It will be appreciated that in other examples, the HMD device900 may take other suitable forms in which a transparent,semi-transparent, and/or non-transparent display is supported in frontof a viewer's eye or eyes. It will also be appreciated that the HMDdevice shown in FIGS. 1-6 may take the form of the HMD device 900, asdescribed in more detail below, or any other suitable HMD device.

The HMD device 900 includes a display system 904 and a see-through ortransparent display 902 that enables images such as holographic objectsto be delivered to the eyes of a wearer of the HMD device. Thetransparent display 902 may be configured to visually augment anappearance of a real-world, physical environment to a wearer viewing thephysical environment through the transparent display. For example, theappearance of the physical environment may be augmented by graphicalcontent (e.g., one or more pixels each having a respective color andbrightness) that is presented via the transparent display 902 to createa mixed reality environment.

The transparent display 902 may also be configured to enable a wearer ofthe HMD device to view a physical, real-world object in the physicalenvironment through one or more partially transparent pixels that aredisplaying a virtual object representation. As shown in FIG. 9, in oneexample the transparent display 902 may include image-producing elementslocated within lenses 906 (such as, for example, a see-through OrganicLight-Emitting Diode (OLED) display). As another example, thetransparent display 902 may include a light modulator on an edge of thelenses 906. In this example, the lenses 906 may serve as a light guidefor delivering light from the light modulator to the eyes of a wearer.Such a light guide may enable a wearer to perceive a 3D holographicimage located within the physical environment that the wearer isviewing, while also allowing the wearer to view physical objects in thephysical environment, thus creating a mixed reality environment.

The HMD device 900 may also include various sensors and related systems.For example, the HMD device 900 may include a gaze tracking system 908that includes one or more image sensors configured to acquire image datain the form of gaze tracking data from a wearer's eyes. Provided thewearer has consented to the acquisition and use of this information, thegaze tracking system 908 may use this information to track a positionand/or movement of the wearer's eyes.

In one example, the gaze tracking system 908 includes a gaze detectionsubsystem configured to detect a direction of gaze of each eye of awearer. The gaze detection subsystem may be configured to determine gazedirections of each of a wearer's eyes in any suitable manner. Forexample, the gaze detection subsystem may comprise one or more lightsources, such as infrared light sources, configured to cause a glint oflight to reflect from the cornea of each eye of a wearer. One or moreimage sensors may then be configured to capture an image of the wearer'seyes.

Images of the glints and of the pupils as determined from image datagathered from the image sensors may be used to determine an optical axisof each eye. Using this information, the gaze tracking system 908 maythen determine a direction the wearer is gazing. The gaze trackingsystem 908 may additionally or alternatively determine at what physicalor virtual object the wearer is gazing. Such gaze tracking data may thenbe provided to the HMD device 900.

It will also be understood that the gaze tracking system 908 may haveany suitable number and arrangement of light sources and image sensors.For example and with reference to FIG. 9, the gaze tracking system 908of the HMD device 900 may utilize at least one inward facing sensor 909.

The HMD device 900 may also include sensor systems that receive physicalenvironment data from the physical environment. For example, the HMDdevice 900 may also include a head tracking system 910 that utilizes oneor more pose sensors, such as pose sensors 912 on HMD device 900, tocapture head pose data and thereby enable position tracking,direction/location and orientation sensing, and/or motion detection ofthe wearer's head. Accordingly, as described above, the tracking system700 of FIG. 7 may receive sensor information from pose sensors thatenable the orientation of the HMD device 900 to be estimated in threedegrees of freedom or the location and orientation of the HMD device tobe estimated in six degrees of freedom.

In one example, head tracking system 910 may comprise an inertialmeasurement unit (IMU) configured as a three-axis or three-degree offreedom position sensor system. This example position sensor system may,for example, include three gyroscopes to indicate or measure a change inorientation of the HMD device 900 within 3D space about three orthogonalaxes (e.g., x, y, z) (e.g., roll, pitch, yaw). The orientation derivedfrom the sensor signals of the IMU may be used to display, via thesee-through display, one or more virtual objects with a body-lockedposition in which the position of each virtual object appears to befixed relative to the wearer of the see-through display and the positionof each virtual object appears to be moveable relative to real-worldobjects in the physical environment.

In another example, head tracking system 910 may comprise an inertialmeasurement unit configured as a six-axis or six-degree of freedomposition sensor system. This example position sensor system may, forexample, include three accelerometers and three gyroscopes to indicateor measure a change in location of the HMD device 900 along the threeorthogonal axes and a change in device orientation about the threeorthogonal axes.

The head tracking system 910 may also support other suitable positioningtechniques, such as GPS or other global navigation systems. Further,while specific examples of position sensor systems have been described,it will be appreciated that any other suitable position sensor systemsmay be used. For example, head pose and/or movement data may bedetermined based on sensor information from any combination of sensorsmounted on the wearer and/or external to the wearer including, but notlimited to, any number of gyroscopes, accelerometers, inertialmeasurement units, GPS devices, barometers, magnetometers, cameras(e.g., visible light cameras, infrared light cameras, time-of-flightdepth cameras, structured light depth cameras, etc.), communicationdevices (e.g., WIFI antennas/interfaces), etc.

In some examples, the HMD device 900 may also include an optical sensorsystem that utilizes one or more outward facing sensors, such as opticalsensor 914 on HMD device 900, to capture image data. The outward facingsensor(s) may detect movements within its field of view, such asgesture-based inputs or other movements performed by a wearer or by aperson or physical object within the field of view. The outward facingsensor(s) may also capture 2D image information and depth informationfrom the physical environment and physical objects within theenvironment. For example, the outward facing sensor(s) may include adepth camera, a visible light camera, an infrared light camera, and/or aposition tracking camera.

The optical sensor system may include a depth tracking system thatgenerates depth tracking data via one or more depth cameras. In oneexample, each depth camera may include left and right cameras of astereoscopic vision system. Time-resolved images from one or more ofthese depth cameras may be registered to each other and/or to imagesfrom another optical sensor such as a visible spectrum camera, and maybe combined to yield depth-resolved video.

In other examples, a structured light depth camera may be configured toproject a structured infrared illumination, and to image theillumination reflected from a scene onto which the illumination isprojected. A depth map of the scene may be constructed based on spacingsbetween adjacent features in the various regions of an imaged scene. Instill other examples, a depth camera may take the form of atime-of-flight depth camera configured to project a pulsed infraredillumination onto a scene and detect the illumination reflected from thescene. For example, illumination may be provided by an infrared lightsource 916. It will be appreciated that any other suitable depth cameramay be used within the scope of the present disclosure.

The outward facing sensor(s) may capture images of the physicalenvironment in which a wearer of the HMD device is situated. Withrespect to the HMD device 900, in one example a mixed reality displayprogram may include a 3D modeling system that uses such captured imagesto generate a virtual environment that models the physical environmentsurrounding the wearer of the HMD device. In some embodiments, theoptical sensor may cooperate with the IMU to determine the location andthe orientation of the head-mounted display device in six degrees offreedom. Such location and orientation information may be used todisplay, via the see-through display, one or more virtual objects with aworld-locked position in which a position of each virtual object appearsto be fixed relative to real-world objects viewable through thesee-through display and the position of each virtual object appears tobe moveable relative to a wearer of the see-through display.

The HMD device 900 may also include a microphone system that includesone or more microphones, such as microphone 918 on HMD device 900, thatcapture audio data. In other examples, audio may be presented to thewearer via one or more speakers, such as speaker 920 on the HMD device900.

The HMD device 900 may also include a controller, such as controller 922on the HMD device 900. The controller may include a logic machine and astorage machine, as discussed in more detail below with respect to FIG.10, that are in communication with the various sensors and systems ofthe HMD device and display. In one example, the storage subsystem mayinclude instructions that are executable by the logic subsystem toreceive signal inputs from the sensors, determine a pose of the HMDdevice 900, and adjust display properties for content displayed on thetransparent display 902.

In some embodiments, the methods and processes described herein may betied to a computing system of one or more computing devices. Inparticular, such methods and processes may be implemented as acomputer-application program or service, an application-programminginterface (API), a library, and/or other computer-program product.

FIG. 10 schematically shows a non-limiting embodiment of a computingsystem 1000 that can enact one or more of the methods and processesdescribed above. Computing system 1000 is shown in simplified form.Computing system 1000 may take the form of one or more head-mounteddisplay devices, or one or more devices cooperating with a head-mounteddisplay device (e.g., personal computers, server computers, tabletcomputers, home-entertainment computers, network computing devices,gaming devices, mobile computing devices, mobile communication devices(e.g., smart phone), and/or other computing devices).

Computing system 1000 includes a logic machine 1002 and a storagemachine 1004. Computing system 1000 may optionally include a displaysubsystem 1006, input subsystem 1008, communication subsystem 1010,and/or other components not shown in FIG. 10.

Logic machine 1002 includes one or more physical devices configured toexecute instructions. For example, the logic machine may be configuredto execute instructions that are part of one or more applications,services, programs, routines, libraries, objects, components, datastructures, or other logical constructs. Such instructions may beimplemented to perform a task, implement a data type, transform thestate of one or more components, achieve a technical effect, orotherwise arrive at a desired result.

The logic machine may include one or more processors configured toexecute software instructions. Additionally or alternatively, the logicmachine may include one or more hardware or firmware logic machinesconfigured to execute hardware or firmware instructions. Processors ofthe logic machine may be single-core or multi-core, and the instructionsexecuted thereon may be configured for sequential, parallel, and/ordistributed processing. Individual components of the logic machineoptionally may be distributed among two or more separate devices, whichmay be remotely located and/or configured for coordinated processing.Aspects of the logic machine may be virtualized and executed by remotelyaccessible, networked computing devices configured in a cloud-computingconfiguration.

Storage machine 1004 includes one or more physical devices configured tohold machine-readable instructions executable by the logic machine toimplement the methods and processes described herein. When such methodsand processes are implemented, the state of storage machine 1004 may betransformed—e.g., to hold different data.

Storage machine 1004 may include removable and/or built-in devices.Storage machine 1004 may include optical memory (e.g., CD, DVD, HD-DVD,Blu-Ray Disc, etc.), semiconductor memory (e.g., RAM, EPROM, EEPROM,etc.), and/or magnetic memory (e.g., hard-disk drive, floppy-disk drive,tape drive, MRAM, etc.), among others. Storage machine 1004 may includevolatile, nonvolatile, dynamic, static, read/write, read-only,random-access, sequential-access, location-addressable,file-addressable, and/or content-addressable devices.

It will be appreciated that storage machine 1004 includes one or morephysical devices. However, aspects of the instructions described hereinalternatively may be propagated by a communication medium (e.g., anelectromagnetic signal, an optical signal, etc.) that is not held by aphysical device for a finite duration.

Aspects of logic machine 1002 and storage machine 1004 may be integratedtogether into one or more hardware-logic components. Such hardware-logiccomponents may include field-programmable gate arrays (FPGAs), program-and application-specific integrated circuits (PASIC/ASICs), program- andapplication-specific standard products (PSSP/ASSPs), system-on-a-chip(SOC), and complex programmable logic devices (CPLDs), for example.

When included, display subsystem 1006 may be used to present a visualrepresentation of data held by storage machine 1004. This visualrepresentation may take the form of a graphical user interface (GUI). Asthe herein described methods and processes change the data held by thestorage machine, and thus transform the state of the storage machine,the state of display subsystem 1006 may likewise be transformed tovisually represent changes in the underlying data. Display subsystem1006 may include one or more display devices utilizing virtually anytype of technology, such as displays 902 of the HMD device 900 shown inFIG. 9. Such display devices may be combined with logic machine 1002and/or storage machine 1004 in a shared enclosure, or such displaydevices may be peripheral display devices.

When included, input subsystem 1008 may comprise or interface with oneor more user-input devices such as a keyboard, mouse, touch screen, orgame controller. In some embodiments, the input subsystem may compriseor interface with selected natural user input (NUI) componentry. Suchcomponentry may be integrated or peripheral, and the transduction and/orprocessing of input actions may be handled on- or off-board. Example NUIcomponentry may include a microphone for speech and/or voicerecognition; an infrared, color, stereoscopic, and/or depth camera formachine vision and/or gesture recognition; a head tracker, eye tracker,accelerometer, and/or gyroscope for motion detection and/or intentrecognition; electric-field sensing componentry for assessing brainactivity; any of the sensors described above with respect to headtracking system 910 of FIG. 9; and/or any other suitable sensor.

When included, communication subsystem 1010 may be configured tocommunicatively couple computing system 1000 with one or more othercomputing devices. Communication subsystem 1010 may include wired and/orwireless communication devices compatible with one or more differentcommunication protocols. As non-limiting examples, the communicationsubsystem may be configured for communication via a wireless telephonenetwork, or a wired or wireless local- or wide-area network. In someembodiments, the communication subsystem may allow computing system 1000to send and/or receive messages to and/or from other devices via anetwork such as the Internet.

It will be understood that the configurations and/or approachesdescribed herein are exemplary in nature, and that these specificembodiments or examples are not to be considered in a limiting sense,because numerous variations are possible. The specific routines ormethods described herein may represent one or more of any number ofprocessing strategies. As such, various acts illustrated and/ordescribed may be performed in the sequence illustrated and/or described,in other sequences, in parallel, or omitted. Likewise, the order of theabove-described processes may be changed.

The subject matter of the present disclosure includes all novel andnon-obvious combinations and sub-combinations of the various processes,systems and configurations, and other features, functions, acts, and/orproperties disclosed herein, as well as any and all equivalents thereof.

The invention claimed is:
 1. A method of operating a see-throughdisplay, the method comprising: displaying via the see-through displayone or more virtual objects with a world-locked position in which aposition of each virtual object appears to be fixed relative toreal-world objects viewable through the see-through display and theposition of each virtual object appears to be moveable relative to awearer of the see-through display; and responsive to transitioning fromtracking a location and an orientation of the see-through display in sixdegrees of freedom to tracking the orientation of the see-throughdisplay in three degrees of freedom due to the location and orientationof the head-mounted display device in six degrees of freedom not meetinga quality metric, displaying via the see-through display the one or morevirtual objects with a body-locked position in which the position ofeach virtual object appears to be fixed relative to the wearer of thesee-through display and the position of each virtual object appears tobe moveable relative to the real-world objects.
 2. The method of claim1, further comprising: visually transitioning the one or more virtualobjects over a series of image frames from the world-locked position tothe body-locked position via an animation.
 3. The method of claim 2,wherein the one or more virtual objects are a plurality of virtualobjects and the plurality of virtual objects visually transition to apredefined layout in the body-locked position.
 4. The method of claim 1,further comprising: visually transitioning the one or more virtualobjects from a 3D view of the one or more virtual objects to a 2D viewof the one or more virtual objects.
 5. The method of claim 4, whereinthe one or more virtual objects are displayed as volumetric holograms inthe 3D view and the one or more virtual objects are displayed in ascreen viewport of a virtual world in the 2D view.
 6. The method ofclaim 1, wherein the transition event includes the see-through displaymoving from a visually recognized real-world location to a visuallyunrecognized real-world location.
 7. The method of claim 6, wherein theone or more virtual objects are a plurality of virtual objects and theplurality of virtual objects is displayed according to alocation-specific visual layout corresponding to the recognizedreal-world location.
 8. A method of operating a see-through display, themethod comprising: displaying via the see-through display one or morevirtual objects with a body-locked position in which a position of eachvirtual object appears to be fixed relative to a wearer of thesee-through display and the position of each virtual object appears tobe moveable relative to real-world objects viewable through thesee-through display; and responsive to transitioning from tracking alocation and an orientation of the see-through display in six degrees offreedom to tracking the orientation of the see-through display in threedegrees of freedom due to the location and orientation of thehead-mounted display device in six degrees of freedom not meeting aquality metric, displaying via the see-through display the one or morevirtual objects with a world-locked position in which the position ofeach virtual object appears to be fixed relative to the real-worldobjects and the position of each virtual object appears to be moveablerelative to the wearer of the see-through display.
 9. The method ofclaim 8, further comprising: visually transitioning the one or morevirtual objects over a series of image frames from the body-lockedposition to the world-locked position via an animation.
 10. The methodof claim 9, wherein the one or more virtual objects are a plurality ofvirtual objects and the plurality of virtual objects visually transitionto a predefined layout in the body-locked position.
 11. The method ofclaim 8, further comprising: visually transitioning the one or morevirtual objects from a 2D view of the one or more virtual objects to a3D view of the one or more virtual objects.
 12. The method of claim 11,wherein the one or more virtual objects are displayed in a screenviewport of a virtual world in the 2D view and the one or more virtualobjects are displayed as volumetric holograms in the 3D view.
 13. Themethod of claim 8, wherein the transition event includes the see-throughdisplay moving from a visually unrecognized real-world location to avisually recognized real-world location.
 14. The method of claim 13,wherein the one or more virtual objects are a plurality of virtualobjects and the plurality of virtual objects is displayed according to alocation-specific visual layout corresponding to the recognizedreal-world location.
 15. A head-mounted display device comprising: aninertial measurement unit (IMU) configured to determine an orientationof the head-mounted display device in three degrees of freedom; anoptical sensor configured to capture image data of a real-worldenvironment and cooperate with the IMU to determine a location and theorientation of the head-mounted display device in six degrees offreedom; a see-through display through which a real-world environment isviewable by a wearer, the see-through display being configured todisplay virtual objects to the wearer such that virtual objects appearto be integrated with the real-world environment, the see-throughdisplay being configured to: responsive to a transition from trackingthe orientation of the head-mounted display device in three degrees offreedom to tracking the location and the orientation of the see-throughdisplay in six degrees of freedom, display via the see-through displayone or more virtual objects with a world-locked position in which aposition of each virtual object appears to be fixed relative toreal-world objects viewable through the see-through display and theposition of each virtual object appears to be moveable relative to awearer of the see-through display, the world-locked position beingderived from the location and orientation of the head-mounted displaydevice in six degrees of freedom; and responsive to a transition fromtracking the location and the orientation of the head-mounted displaydevice in six degrees of freedom to tracking the orientation of thesee-through display in three degrees of freedom, displaying via thesee-through display the one or more virtual objects with a body-lockedposition in which the position of each virtual object appears to befixed relative to the wearer of the see-through display and the positionof each virtual object appears to be moveable relative to the real-worldobjects, the body-locked position being derived from the orientation ofthe head-mounted display device in three degrees of freedom.
 16. Thehead-mounted display device of claim 15, wherein the transition fromtracking the orientation of the head-mounted display device in threedegrees of freedom to tracking the location and the orientation of thesee-through display in six degrees of freedom is performed responsive tothe location and the orientation of the head-mounted display device insix degrees of freedom meeting the quality metric, and the transitionfrom tracking the location and the orientation of the head-mounteddisplay device in six degrees of freedom to tracking the orientation ofthe see-through display in three degrees of freedom is performedresponsive to the location and orientation of the head-mounted displaydevice in six degrees of freedom not meeting the quality metric.